Skip Headers

Oracle® Universal Installer and OPatch User's Guide 10g Release 2 (10.2) for Windows and UNIX Part Number B16227-02 |
Home | Book List | Contents | Index | Master Index | Feedback

Previous | Next
| View PDF

# 8 Oracle Software Patching Using OPatch

The OPatch utility is a tool that allows the application and rollback of interim patches to Oracle products. This chapter provides information on using OPatch to apply patches.This chapter includes the following topics:

- About Interim Patches

- Structure of Interim Patches

- About OPatch

- Requirements for OPatch

- Features of OPatch

- Backup and Recovery Considerations for Patching

- Operations and Options in OPatch Utility

- Real Application Clusters Patching

- Recovering from a Failed Patching Session Using OPatch

- Troubleshooting OPatch

## About Interim Patches

Patches are a small collection of files that are copied over an existing installation. They are associated to particular versions of Oracle Products. Patches, when applied to the correct version of an installed product, results in an upgraded version of the product.Interim patches are bug fixes that are made available to customers in response to specific bugs. They require a particular base release or patchset to be installed before they can be applied. They generally address specific bugs for a particular customer. These patches are not versioned and are generally made available in a future patchset as well as the next product release.

## Structure of Interim Patches

Interim patches generally come in a zipped format. You need to unzip them before you apply a patch. The following figure illustrates the structure of the interim patch.

Figure 8-1 Structure of Interim Patches

The interim patches have the following contents:

- Patch Metadata: This contains information on the patch ID, the bugs fixed, the files affected, and the actions to be performed.

- Payload: This contains the files that will be modified by OPatch.

- Custom Scripts: Pre-processing and post-processing scripts that needs to be run before and after patching.

## Interim Patch Versions

OPatch 10.2 supports maintaining versions of patches. You can have two or more different versions of the same patch (with the same patch ID). This version information is stored in the OPatch metadata. The metadata has a tag `date_of_patch`, that stores the patch version information. The sample of the tag is as follows:

    <date_of_patch year="YYYY" month="mmm" day="Day" time="Time" zone="TimeZone"/>: Date on which the patch was created

This tag records the time of creation of the patch by Oracle. If the same patch is created at a later point of time, this tag will record that time.

Note: This version information is the time of creation of the patch by Oracle and not the time of application of the patch in the host.

For example, consider a patch with ID 300200 and the `date_of_patch` tag in the patch metadata is as follows:

    <date_of_patch year="2003" month="Dec" day="24" time="04:57:13 hrs" zone="US/Eastern"/>

OPatch will consider this version of the patch to be created on December 24th, 2003 at 04:57:13 hrs.When you apply an interim patch to an Oracle home, OPatch stores the patch information in `$ORACLE_HOME/.patch_storage` directory. Inside this directory, there are separate directories created for each patch applied to the Oracle home. You can only have one version of the patch applied in the system at a given time.

You can determine the location of the patch information directory by executing the `opatch lsinventory -detail` command and looking for the patch location storage area information in the output. The sample is as follows:

    Patch Location in Storage area:
    /home1/HOMEtoiir571/.patch_storage/300200_Dec_24_2003_04_57_13

You will also find an unzipped version of the patch in the following location:

    $ORACLE_HOME/.patch_storage/<patch_id_timestamp>/original_patch

## Getting Interim Patches

Oracle releases interim patches frequently to fix a bug or a set of bugs. You can get the interim patches by specifying the patch ID in OracleMetalink from the following location:

    http://www.oracle.com/support/metalink/index.html

## About OPatch

OPatch is an Oracle supplied utility to assist you with the process of applying interim patches to Oracle's software. OPatch is a Java-based utility which requires the Oracle Universal Installer to be installed. It is platform independent and runs on all supported operating systems.

OPatch supports the following:

- Applying an interim patch.

- Rolling back the application of an interim patch.

- Detecting conflict when applying an interim patch after previous interim patches have been applied. It also suggests the best options to resolve a conflict.

- Reporting on installed products and interim patch.

## Requirements for OPatch

The OPatch utility requires the following:

- The Oracle home environment variable (`ORACLE_HOME`) must point to a valid Oracle home directory and match the value used during installation of the Oracle home directory.

- Java SDK 1.4 or higher, Java commands for Windows and ar, cp, fuser, and, make commands for UNIX must be made available.

- The library path must be set correctly for Oracle Real Application Clusters environments. OPatch uses some APIs to detect if the system is Real Application Clusters. Ensure that the library path is set correctly as follows:

      For Solaris
      LD_LIBRARY_PATH = $ORACLE_HOME/lib32:$ORACLE_HOME/lib
      For HP-UX -
      SHLIB_PATH=$ORACLE_HOME/lib32:/usr/lib

  See Also: For the latest information about the OPatch utility, and to check for updates, and to get latest versions refer to OracleMetaLink at http://www.oracle.com/support/metalink/index.html

## Features of OPatch

The OPatch 10.2 utility has the following features:

- Scalability: OPatch is scalable to support large number of patches.

- Reliability: OPatch is reliable and protects the Oracle home and inventory. It can bring the Oracle home back to a stable state from patch application failures. It can also easily detect patch conflicts.

- Portability: OPatch is compatible with all operating systems for which Oracle releases software.

- Robust: OPatch is very robust. It is very easy to apply a patch as well as remove it.

- Easy to maintain: OPatch is easy to maintain and is also extensible.

- Support for Silent Operation: OPatch supports silent operation. This mode allows you to run the software without any user interaction.

- Support for Real Application Clusters: OPatch supports Real Application Clusters and works well in that setup. It is easy to extend it to the Grid Control.

- Easy to debug: OPatch has various levels of logging and tracing mechanisms. It also has a debug option that helps to diagnose problems with the software easily.

## Pre-requisite Checks in OPatch

Before you invoke OPatch, perform the following pre-requisite checks:

### Check ORACLE_HOME and Environment Variable

OPatch verifies if the Oracle home is present. You must ensure that the `ORACLE_HOME` environment variable is set to the Oracle home of the product you are trying to patch. Check the respective vendor documentation for the details to set the environment variable.

### Check for JRE and JDK

OPatch requires JDK 1.4 or higher to work properly. JRE comes as a part of JDK.

OPatch 10.2 uses the jar utility that comes with JDK for its jar, war, and ear operations. Opatch will look for JDK inside the Oracle home specified. In case the Oracle home does not have JDK, the user has to use the `-jdk` option in OPatch to provide an alternate location. OPatch will display an error, if there is a jar/war/ear operation and is unable to locate Java SDK location.

In order to find the version of JDK installed, execute the following command:

    JDK/bin/java -version

Note: If the patch you are applying does not have a jar action, you might not need JDK and OPatch will work fine with JRE alone.

### Check for System Space

When OPatch processes the script for the installation of a patch, it simultaneously generates a rollback script and saves a copy of every file edited or deleted during the patching. OPatch also backs up the inventory information. So, Oracle recommends that you have sufficient system space to accommodate the patch and the backup information.

### Check for Oracle Universal Installer and OPatch Version Compatibility

OPatch 10.2 requires Oracle Universal Installer 10.2 or higher to work properly. If the Oracle Universal Installer version is less than what OPatch requires, then OPatch errors out.

### Check for Patch Applicable on Operating System

OPatch detects if a particular patch is applicable for an operating system. If it is not applicable, it gives out a suitable error message.

### Check for System Commands

OPatch supports a set of properties that are used for various operations of the software. You can use these properties to control the internal operations of OPatch. By default, OPatch uses standard Java property format to specify the properties. An exhaustive list of the default properties and their values are as follows:

    fuser=/sbin:/usr/sbin

    ar=/usr/ccs/bin/

    make=/usr/bin

You can specify OPatch properties in the following ways:

- By using the default OPatch properties.

- By specifying the location of the user-defined properties file.

- By using the command line. The syntax is as follows:

      PROPERTY_NAME=VALUE

  Example: fuser=/sbin:/usr/sbin

### Additional Pre-requisite Checks for Real Application Clusters

For a Real Application Clusters, ensure that you perform the following pre-requisite checks in addition to the other checks listed in the preceding section:

#### Check for User Equivalence

You must ensure that the cluster machines should have user equivalence set for the user installing Oracle Clusterware/ Real Application Clusters. On UNIX, this means rsh or ssh or both should be setup on the cluster machines. On Windows, this means the same `<domain>\<user>` should have administrative privileges on all the cluster machines and the machines should be a member of the `<domain>`.

If the user equivalence is set properly, the following command will work properly:

    $ rsh <nodename> date

For more information on setting user equivalence, refer to section "Configuring SSH on all Cluster Nodes".

#### Check for OPatch lsinventory

Ensure that you are able to invoke `opatch lsinventory -detail` command and are able to see the node information being printed out. A sample listing of the output of the command is as follows:

    Oracle interim Patch Installer version 10.2.0.1.0
    Copyright (c) 2005, Oracle Corporation.  All rights reserved..
    Oracle Home       : /home/racqa/102_twork/toii/toiir/toiir571/HOMEtoiir571
    Central Inventory : /home/racqa/102_twork/toii/toiir/toiir571/inventory
       from           : /home/racqa/102_
    twork/toii/toiir/toiir571/HOMEtoiir571/oraInst.loc
    OPatch version    : 10.2.0.1.0
    OUI version       : 10.2.0.1.0
    OUI location      : /home/racqa/102_twork/toii/toiir/toiir571/HOMEtoiir571/oui
    Log file location : /home/racqa/102_
    twork/toii/toiir/toiir571/HOMEtoiir571/cfgtoollogs/opatch/opatch-2005_May_30_
    01-04-52-PDT_Mon.log
    Lsinventory Output file location : /home1/racqa/102_
    twork/toii/toiir/toiir571/HOMEtoiir571/cfgtoollogs/opatch/lsinv/lsinventory-2005_
    May_30_01-04-52-PDT_Mon.txt
    --------------------------------------------------------------------------------
    Installed Top-level Products (1):
    Oracle Database 10g                                                  10.1.0.2.0
    There are 1 products installed in this Oracle Home.
    Installed Products (151):
    Advanced Queueing (AQ) API                                           10.1.0.2.0
    Advanced Replication                                                 10.1.0.2.0
    Agent Required Support Files                                         10.1.0.2.0
    Assistant Common Files                                               10.1.0.2.0
    Authentication and Encryption                                        10.1.0.2.0
    Bali Share                                                           1.1.18.0.0
    Character Set Migration Utility                                      10.1.0.2.0
    CSS Single-instance Common Files                                     10.1.0.2.0
    Data Management Services Common Files                                10.1.0.2.0
    Database Configuration Assistant                                     10.1.0.2.0
    Database SQL Scripts                                                 10.1.0.2.0
    Database Upgrade Assistant                                           10.1.0.2.0
    Database Verify Utility                                              10.1.0.2.0
    Database Workspace Manager                                           10.1.0.2.0
    DBJAVA Required Support Files                                        10.1.0.2.0
    Documentation Required Support Files                                 10.1.0.2.0
    Enterprise Edition Options                                           10.1.0.2.0
    Enterprise Manager Agent                                             10.1.0.2.0
    Enterprise Manager Common Files                                      10.1.0.2.0
    Enterprise Manager Minimal Integration                               10.1.0.2.0
    Enterprise Manager plugin Common Files                               10.1.0.2.0
    Enterprise Manager Repository                                        10.1.0.2.0
    Export/Import                                                        10.1.0.2.0
    Extended Windowing Toolkit                                           3.3.18.0.0
    Generic Connectivity Common Files                                    10.1.0.2.0
    Generic Connectivity Using ODBC                                      10.1.0.2.0
    Installation Common Files                                            10.1.0.2.0
    Installer SDK Component                                              10.2.0.1.0
    iSQL*Plus                                                            10.1.0.2.0
    Java Naming and Directory Interface Libraries                        10.1.0.2.0
    Java Runtime Environment                                              1.4.2.0.0
    JDBC Common Files                                                    10.1.0.2.0
    JDBC/OCI Common Files                                                10.1.0.2.0
    JDBC/OCI Common Files for Instant Client                             10.1.0.2.0
    LDAP Required Support Files                                          10.1.0.2.0
    New Database ID                                                      10.1.0.2.0
    Object Type Translator                                               10.1.0.2.0
    Oracle Advanced Security                                             10.1.0.2.0
    Oracle C++ Call Interface                                            10.1.0.2.0
    Oracle C++ Call Interface for Instant Client                         10.1.0.2.0
    Oracle Call Interface (OCI)                                          10.1.0.2.0
    Oracle Client Required Support Files                                 10.1.0.2.0
    Oracle Code Editor                                                   1.2.1.0.0I
    Oracle Containers for Java                                           10.1.0.2.0
    Oracle Core Required Support Files                                   10.1.0.2.0
    Oracle Data Mining                                                   10.1.0.2.0
    Oracle Database 10g                                                  10.1.0.2.0
    Oracle Database 10g                                                  10.1.0.2.0
    Oracle Database User Interface                                       2.2.13.0.0
    Oracle Database Utilities                                            10.1.0.2.0
    Oracle Development Kit                                               10.1.0.2.0
    Oracle Display Fonts                                                  9.0.2.0.0
    Oracle Enterprise Manager 10g Database Control                       10.1.0.2.0
    Oracle Enterprise Manager Console DB                                 10.1.0.2.0
    Oracle Extended Windowing Toolkit                                    3.4.28.0.0
    Oracle Globalization Support                                         10.1.0.2.0
    Oracle Help For Java                                                 4.2.5.0.0a
    Oracle Help for the  Web                                             1.1.7.0.0a
    Oracle Ice Browser                                                    5.2.3.3.0
    Oracle interMedia                                                    10.1.0.2.0
    Oracle interMedia Annotator                                          10.1.0.2.0
    Oracle interMedia Audio                                              10.1.0.2.0
    Oracle interMedia Client Compatibility Files                         10.1.0.2.0
    Oracle interMedia Client Demos                                       10.1.0.2.0
    Oracle interMedia Client Option                                      10.1.0.2.0
    Oracle interMedia Common Files                                       10.1.0.2.0
    Oracle interMedia Image                                              10.1.0.2.0
    Oracle interMedia Java Advanced Imaging                              10.1.0.2.0
    Oracle interMedia Java Client                                        10.1.0.2.0
    Oracle interMedia Locator                                            10.1.0.2.0
    Oracle interMedia Video                                              10.1.0.2.0
    Oracle interMedia Web Client                                         10.1.0.2.0
    Oracle Internet Directory Client                                     10.1.0.2.0
    Oracle Internet Directory Client Common Files                        10.1.0.2.0
    Oracle Internet Directory Tools                                      10.1.0.2.0
    Oracle Java Tools                                                    10.1.0.2.0
    Oracle JDBC Thin Driver for JDK 1.2                                  10.1.0.2.0
    Oracle JDBC Thin Driver for JDK 1.4                                  10.1.0.2.0
    Oracle JDBC Thin Driver for JDK 1.4 for Instant Client               10.1.0.2.0
    Oracle JDBC/OCI Driver for JDK 1.4                                   10.1.0.2.0
    Oracle JFC Extended Windowing Toolkit                                4.2.18.0.0
    Oracle JVM                                                           10.1.0.2.0
    Oracle Locale Builder                                                10.1.0.2.0
    Oracle Message Gateway Common Files                                  10.1.0.2.0
    Oracle Net                                                           10.1.0.2.0
    Oracle Net Configuration Assistant                                   10.1.0.2.0
    Oracle Net Listener                                                  10.1.0.2.0
    Oracle Net Manager                                                   10.1.0.2.0
    Oracle Net Required Support Files                                    10.1.0.2.0
    Oracle Net Services                                                  10.1.0.2.0
    Oracle Notification Service                                           9.0.4.0.0
    Oracle OLAP                                                          10.1.0.2.0
    Oracle OLAP API                                                      10.1.0.2.0
    Oracle OLAP Catalog                                                  10.1.0.2.0
    Oracle One-Off Patch Installer                                       10.2.0.1.0
    Oracle Partitioning                                                  10.1.0.2.0
    Oracle Programmer                                                    10.1.0.2.0
    Oracle RAC Required Support Files                                    10.1.0.2.0
    Oracle RAC Required Support Files-HAS                                10.1.0.2.0
    Oracle Real Application Clusters                                     10.1.0.2.0
    Oracle Spatial                                                       10.1.0.2.0
    Oracle Starter Database                                              10.1.0.2.0
    Oracle Text                                                          10.1.0.2.0
    Oracle UIX                                                          2.1.21.0.0a
    Oracle Ultra Search Common Files                                     10.1.0.2.0
    Oracle Ultra Search Middle-Tier                                      10.1.0.2.0
    Oracle Ultra Search Server                                           10.1.0.2.0
    Oracle Universal Installer                                           10.2.0.1.0
    Oracle Wallet Manager                                                10.1.0.2.0
    Oracle XML Developer's Kit                                           10.1.0.2.0
    Oracle XML Runtime Components                                        10.1.0.2.0
    Oracle XML SQL Utility                                               10.1.0.2.0
    Oracle10g Real Application Clusters Common Files                     10.1.0.2.0
    Parser Generator Required Support Files                              10.1.0.2.0
    Perl Interpreter                                                     5.6.1.0.2d
    PL/SQL                                                               10.1.0.2.0
    PL/SQL Embedded Gateway                                              10.1.0.2.0
    PL/SQL Required Support Files                                        10.1.0.2.0
    Platform Required Support Files                                      10.1.0.2.0
    Precompiler Common Files                                             10.1.0.2.0
    Precompiler Required Support Files                                   10.1.0.2.0
    Pro*C/C++                                                            10.1.0.2.0
    RDBMS Required Support Files                                         10.1.0.2.0
    Recovery Manager                                                     10.1.0.2.0
    regexp                                                                2.1.9.0.0
    Required Support Files                                               10.1.0.2.0
    Sample Schema                                                        10.1.0.2.0
    Secure Socket Layer                                                  10.1.0.2.0
    Secure Socket Layer                                                  10.1.0.2.0
    SQL*Loader                                                           10.1.0.2.0
    SQL*Plus                                                             10.1.0.2.0
    SQL*Plus Required Support Files                                      10.1.0.2.0
    SQLJ Runtime                                                         10.1.0.2.0
    SSL Required Support Files                                           10.1.0.2.0
    SSL Required Support Files for InstantClient                         10.1.0.2.0
    Sun JDK                                                               1.4.2.0.0
    Sun JDK extensions                                                    9.0.4.0.0
    Utilities Common Files                                               10.1.0.2.0
    Visigenics ORB                                                        3.4.0.0.0
    XDK Required Support Files                                           10.1.0.2.0
    XML                                                                  10.1.0.2.0
    XML Class Generator for C++                                          10.1.0.2.0
    XML Class Generator for Java                                         10.1.0.2.0
    XML Parser for C                                                     10.1.0.2.0
    XML Parser for C++                                                   10.1.0.2.0
    XML Parser for Java                                                  10.1.0.2.0
    XML Parser for Oracle JVM                                            10.1.0.2.0
    XML Parser for PL/SQL                                                10.1.0.2.0
    XML Transviewer Beans                                                10.1.0.2.0
    XML Transx                                                           10.1.0.2.0
    XSQL Servlet                                                         10.1.0.2.0
    There are 151 products installed in this Oracle Home.
    Intermin patches (1) :
    Patch  3811942      : applied on Mon May 30 00:59:33 PDT 2005
       Created on 31 Aug 2004, 12:06:28 hrs US/Pacific
       Bugs fixed:
         3811942
       Files Touched:
         /hosp.o --> ORACLE_HOME/lib/libagent10.a
         /pesblt.o --> ORACLE_HOME/lib/libpls10.a
         /kgl.o --> ORACLE_HOME/lib/libgeneric10.a
         /qcpi6.o --> ORACLE_HOME/lib/libgeneric10.a
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iextjob
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iextjobo
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/client_sharedlib
         /hosp.o --> ORACLE_HOME/lib/libserver10.a
         /prse.o --> ORACLE_HOME/lib/libserver10.a
         /prsa.o --> ORACLE_HOME/lib/libserver10.a
         /prsf.o --> ORACLE_HOME/lib/libserver10.a
         /prssz.o --> ORACLE_HOME/lib/libserver10.a
         /kprc.o --> ORACLE_HOME/lib/libserver10.a
         /qmhdr.o --> ORACLE_HOME/lib/libserver10.a
         /pesblt.o --> ORACLE_HOME/lib/libpls10.a
         /qcpi6.o --> ORACLE_HOME/lib/libgeneric10.a
         prvtpexp.plb --> ORACLE_HOME/rdbms/admin/prvtpexp.plb
         sjsex.o --> ORACLE_HOME/rdbms/lib/sjsex.o
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iextjob
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iextjobo
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle
         ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/client_sharedlib
         driload.pkh --> ORACLE_HOME/ctx/admin/driload.pkh
         /oracle/sysman/assistants/util/em/EMConfigStep.class --> ORACLE_
    HOME/assistants/jlib/assistantsCommon.jar
         /oracle/sysman/assistants/util/sqlEngine/SQLEngine.class --> ORACLE_
    HOME/assistants/jlib/assistantsCommon.jar
         /oracle/sysman/assistants/dbca/backend/DBEntryStep.class --> ORACLE_
    HOME/assistants/dbca/jlib/dbca.jar
         /oracle/sysman/assistants/dbca/backend/EMConfigStep.class --> ORACLE_
    HOME/assistants/dbca/jlib/dbca.jar
         /oracle/sysman/assistants/dbca/backend/PostDBCreationStep.class --> ORACLE_
    HOME/assistants/dbca/jlib/dbca.jar
         /oracle/sysman/emcp/EMConfig.class --> ORACLE_HOME/jlib/emca.jar
         /oracle/sysman/emcp/IEMCAConstants.class --> ORACLE_HOME/jlib/emca.jar
         /oracle/sysman/emcp/EMConfig.class --> ORACLE_HOME/sysman/jlib/emCORE.jar
         /oracle/sysman/emcp/IEMCAConstants.class --> ORACLE_
    HOME/sysman/jlib/emCORE.jar
         /oracle/sysman/emSDK/conf/ConfigManager.class --> ORACLE_
    HOME/sysman/jlib/emCORE.jar
         /oracle/sysman/emSDK/eml/EmlConstants.class --> ORACLE_
    HOME/sysman/jlib/emCORE.jar
         /oracle/sysman/util/pref/PrefUtil.class --> ORACLE_
    HOME/sysman/jlib/emCORE.jar
         emctl.pl --> ORACLE_HOME/emdw/bin/emctl.pl
         /ncrfipm.o --> ORACLE_HOME/lib/libnro10.a
         libsqlplus.a --> ORACLE_HOME/lib/libsqlplus.a
         libsqlplus.so --> ORACLE_HOME/lib/libsqlplus.so
         libisqlplus.a --> ORACLE_HOME/lib/libisqlplus.a
         libisqlplus.so --> ORACLE_HOME/lib/libisqlplus.so
         mdprivs.sql --> ORACLE_HOME/md/admin/mdprivs.sql
         prvtgmd.plb --> ORACLE_HOME/md/admin/prvtgmd.plb
         sdolrsmd.sql --> ORACLE_HOME/md/admin/sdolrsmd.sql
         prvtccbk.plb --> ORACLE_HOME/md/admin/prvtccbk.plb
         wk0acl.pkh --> ORACLE_HOME/ultrasearch/admin/wk0acl.pkh
         wk0acl.plb --> ORACLE_HOME/ultrasearch/admin/wk0acl.plb
         wk0adm.pkh --> ORACLE_HOME/ultrasearch/admin/wk0adm.pkh
         wk0adm.plb --> ORACLE_HOME/ultrasearch/admin/wk0adm.plb
         wk0snapshot.plb --> ORACLE_HOME/ultrasearch/admin/wk0snapshot.plb
         wk0util.pkh --> ORACLE_HOME/ultrasearch/admin/wk0util.pkh
         wk0util.plb --> ORACLE_HOME/ultrasearch/admin/wk0util.plb
       Patch Location in Inventory:
         /home1/racqa/102_
    twork/toii/toiir/toiir571/HOMEtoiir571/inventory/oneoffs/3811942
       Patch Location in Storage area:
         /home1/racqa/102_twork/toii/toiir/toiir571/HOMEtoiir571/.patch_
    storage/3811942_Aug_31_2004_12_06_28
    Rac system comprising of multiple nodes
      Local node = stadu56
      Remote node = stacg16
    --------------------------------------------------------------------------------
    OPatch succeeded.

If you do not find the node information correctly printed out, you need to update the node list. For more information on updating the node list refer to section "Updating the Nodes of a Cluster".

## Backup and Recovery Considerations for Patching

You can back up the `ORACLE_HOME` using your preferred method. You can use any method such as `zip, cp -r, tar,` and `cpio` to compress the `ORACLE_HOME.`

If the `ORACLE_HOME` does not show up when doing an `opatch lsinventory -detail`, then the `ORACLE_HOME` might be missing from the Central Inventory or the Central Inventory itself could be missing or corrupted.

If the `ORACLE_HOME` is listed when you execute an `opatch lsinventory -detail` command, but the products and components within the `ORACLE_HOME` is not listed, then it could be that the inventory within the `ORACLE_HOME` (local inventory) is missing or corrupted.

If the local inventory is corrupted or lost for some reason, you can simply restore the `ORACLE_HOME/inventory` if it had been backed up. If a backup does not exist, you may have to reinstall the software.

It is highly recommended to back up the `ORACLE_HOME` before any patch operation.

## Operations and Options in OPatch Utility

The OPatch utility is located in the `<Path_to_Oracle_Home>/OPatch` directory. It is run with various options and command-line arguments. The following command shows the syntax for the OPatch utility:

    <Path_to_OPatch>/opatch <option> [-command_line_arguments]

In the preceding command, the following variables are used:

- command_line_arguments: The command-line arguments for the option. They are described in the following sections.

- option: The OPatch option. They are described in the following table:

Table 8-1 OPatch Options

Option | Description
--- | ---
apply | Installs an interim patch. Refer to "apply Option" for more information.
lsinventory | Lists what is currently installed on the system. Refer to "lsinventory Option" for more information.
query | Queries a given patch for specific details. Refer to "query Option" for more information.
rollback | Removes an interim patch. Refer to "rollback Option" for more information.
version | Prints the current version of the patch tool. Refer to "version Option" for more information.

To view additional information for any option, use the following command:

    <Path_to_OPatch>/opatch option -help

If using Perl, then use the following command:

    perl opatch.pl option -help

### apply Option

The `apply` option applies an interim patch to a specified Oracle home. The `ORACLE_HOME` environment variable must be set to the Oracle home to be patched. The following syntax is used for this option:

    <Path_to_OPatch>/opatch apply [-delay (value)] [-force] \
    [-invPtrLoc (path)] [-jdk (location)] [-jre (location)] [-local] \
    [-minimize_downtime] [-no_bug_superset] [-no_inventory] \
    [-oh (Oracle Home location)] \
    [-post (options to be passed into post) [-opatch_post_end]]\
    [-pre (options to be passed into pre) [-opatch_pre_end]] \
    [-retry (value)] [-silent] [-verbose] [-no_relink] \ [-no_sysmod (patch
     location)][-remote_nodes (comma separated node names)][-local_node (node
    _name)][patch_location]

The following table lists the command-line arguments available for use with the `apply` option:

Table 8-2 apply option command line arguments

Command-line Argument | Description
--- | ---
delay | Specifies how many seconds to wait before attempting to lock the inventory in the case of a previous failure. You can use this option only if `-retry` option is specified.
force | Removes conflicting patches from the system. If a conflict exists which prevents the patch from being applied, then the `-force` command-line argument can be used to apply the patch.
invPtrLoc | Specifies the location of the `oraInst.loc` file. This command-line argument is needed when the `-invPtrLoc` argument was used during installation. Oracle recommends the use of the default Central Inventory for a platform.
jdk | Specifies the location of a particular JDK (jar) to use instead of the default location under the Oracle home directory. You cannot use -jdk and -jre options together.
jre | Specifies the location of a particular JRE (Java) to use instead of the default location under the Oracle home directory. You cannot use -jdk and -jre options together.
local | Specifies that the OPatch utility patch the local node and update the inventory of the local node. It does not propagate the patch or inventory update to other nodes. This command-line argument can be used on Oracle Real Application Clusters environments and non-clustered environments. If an entire cluster is shutdown before patching, then this argument can be used for non-rolling patches.
local_node | Specifies to the OPatch utility the local node for this cluster. This command-line argument can be used on Oracle Real Application Clusters environments.
minimize_downtime | Specifies the order of nodes to be patched by the OPatch utility. This command-line argument only applies to Oracle Real Application Clusters environments. It cannot be used with the `-local` command-line argument or a rolling patch.
no_bug_superset | Specifies to error out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory.
no_inventory | Bypasses the inventory for reading and updates. This command-line argument cannot be used with the `-local` command-line argument. This command-line argument puts the installation into an unsupported state.
no_sysmod | Specifies that the OPatch utility need not update the files in the system. It will only update the inventory.
no_relink | This option does not perform any make operation. It can be used during multiple patch applications and to perform the linking step only once. OPatch does not keep track of the make operations it did not perform. You need to make sure to execute OPatch without this option at the end for compilation.
oh | Specifies the Oracle home directory to use instead of the default.
opatch_post_end | Marks the end of the `post` option. This command-line argument is used with the `post` command-line argument. If this argument is not used, then everything after `post` is passed into `post`.
opatch_pre_end | Marks the end of the `pre` options. This command-line argument is used with the `pre` command-line argument. If this argument is not used, then everything after `pre` is passed into `pre`.
post | Specifies the parameters to be passed inside the `post` script besides the standard parameters.
pre | Specifies the parameters to be passed inside the `pre` script besides the standard parameters.
remote_nodes | Specifies to the OPatch utility the list of remote nodes. This command-line argument can be used on Oracle Real Application Clusters environments.
report | Prints the action to the screen without executing it.
retry | Specifies how many times the OPatch utility should try when there is an inventory lock failure.
silent | Suppresses user interaction, and defaults any answers to "yes."
verbose | Prints output to the screen as well as to the log file.

Note: If a patch consists of SQL changes, follow the instructions in the patch readme, included with the patch to apply the SQL scripts.

### lsinventory Option

The `lsinventory` option reports what has been installed on the system for a particular Oracle home directory, or for all installations. The following syntax is used for this option:

The following table lists the command-line arguments available for use with the `lsinventory` option:

    <Path_to_OPatch>/opatch lsinventory [-all] [-delay (value)] [-detail] [-invPrtLoc (path)] \
    [-jre (location)] [-retry (value)] [-patch] [-oh (Oracle Home location)]

Table 8-3 lsinventory option command line arguments

Command-line Argument | Description
--- | ---
all | Reports the name and installation directory for each found Oracle home directory.
delay | Specifies how many seconds to wait before attempting to lock the inventory in the case of a previous failure. You can use this option only if -retry option is specified.
detail | Reports the installed products and other details. This command-line argument cannot be used with the `-all` command-line argument.
invPtrLoc | Specifies the location of the `oraInst.loc` file. This command-line argument is needed when the `invPtrLoc` command-line argument was used during installation. Oracle recommends the use of the default Central Inventory for a platform.
jre | Specifies the location of a particular JRE (Java) to use instead of the default location under the Oracle home directory.
retry | Specifies how many times the OPatch utility should try when there is an inventory lock failure.
oh | Specifies the Oracle home directory to use instead of the default directory.
patch | Specifies the patches installed in the Oracle home.

The following is a sample output of `opatch lsinventory -detail`:

    Oracle interim Patch Installer version 10.2.0.0.0
    Copyright (c) 2005, Oracle Corporation.  All rights reserved..
    Oracle Home       : /home/oracle_TEST/product/10.2.0/db_1
    Central Inventory : /home/OUIHome_Opatch
       from           : /home/oracle_TEST/product/10.2.0/db_1/oraInst.loc
    OPatch version    : 10.2.0.0.0
    OUI version       : 10.2.0.0.0
    OUI location      : /home/oracle_TEST/product/10.2.0/db_1/oui
    Log file location : /home/oracle_
    TEST/product/10.2.0/db1/cfgtoollogs/opatch/opatch-2005_May_25_11-09-34-IST_Wed.log
    Lsinventory Output file location : /home/oracle_TEST/product/10.2.0/db_
    1/cfgtoollogs/opatch/lsinv/lsinventory-2005_May_25_11-09-34-IST_Wed.txt
     --------------------------------------------------------------------------------
    Installed Top-level Products (1):
    Oracle Database 10g                                                  10.2.0.0.0
    There are 1 products installed in this Oracle Home.
    Installed Products (10):
    Agent Required Support Files                                         10.2.0.0.0
    Assistant Common Files                                               10.2.0.0.0
    Bali Share                                                           1.1.18.0.0
    Buildtools Common Files                                              10.2.0.0.0
    Character Set Migration Utility                                      10.2.0.0.0
    Database Configuration and Upgrade Assistants                        10.2.0.0.0
    Database SQL Scripts                                                 10.2.0.0.0
    Database Workspace Manager                                           10.2.0.0.0
    DBJAVA Required Support Files                                        10.2.0.0.0
    Enterprise Edition Options                                           10.2.0.0.0
    There are 10 products installed in this Oracle Home.
    Intermin patches (1) :
    Patch  102000       : applied on Mon May 23 19:44:08 IST 2005
    Created on 27 Jul 2004, 05:43:46 hrs PST8PDT
    Bugs fixed: 102000
    Files Touched:
    /qmtest.o --> ORACLE_HOME/lib/libserver10.a
    libmapsym.so --> ORACLE_HOME/lib/libmapsym.so
    ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle
    /oracle/xml/jaxb/orajaxb.class --> ORACLE_HOME/lib/xml.jar
    Patch Location in Inventory:
    /home/oracle_TEST/product/10.2.0/db_1/inventory/oneoffs/102000
    Patch Location in Storage area:
    /home/oracle_TEST/product/10.2.0/db_1/.patch_storage/102000_Jul_27_2004_05_43_46
     --------------------------------------------------------------------------------
     OPatch succeeded.

### query Option

The `query` option queries a specific patch for specific details. It provides information about the patch and the system being patched. The following syntax is used for this option:

    <Path_to_OPatch>/opatch query [-all] [-jre (Location)] [-jdk (Location)]\
     [-oh (Location)] [patch_location]

The following table lists the command-line arguments available for use with the `query` option:

Table 8-4 query option command line arguments

Command-line Argument | Description
--- | ---
all | Retrieves all information about a patch. This is equivalent to setting all command-line arguments.
jdk | Specifies the location of a particular JDK (jar) to use instead of the default location under the Oracle home directory. You cannot use -jdk and -jre options together.
jre | Specifies the location of a particular JRE (Java) to use instead of the default location under the Oracle home directory. You cannot use -jdk and -jre options together.
oh | Specifies the Oracle home directory to use instead of the default directory.

### rollback Option

The `rollback` option removes a specific interim patch from the appropriate Oracle home directory. The following syntax is used for this option:

    <Path_to_OPatch>/opatch rollback -id (patch_id) [-ph (patch directory)] \
    [-delay] (value) [-invPtrLoc (path)] [-jdk (location)] [-jre (location)]\
    [-local] [-oh (Oracle Home location)] \
    [-post (options to be passed into post) [-opatch_post_end]] \
    [-pre (options to be passed into pre) [-opatch_pre_end]] [-retry (value)] \
    [-silent] [-verbose] [-no_relink] [-no_sysmod][-remote_nodes (node1,node2)][-local_node (node_name)]

The following table lists the command-line arguments available for use with the `rollback` option:

Table 8-5 rollback option command line arguments

Command-line Argument | Description
--- | ---
delay | Specifies how many seconds the OPatch utility should wait before attempting to lock inventory again, if the `-retry` command-line argument is used with the `apply` option.
id | Indicates the patch to be rolled back. Use the `-lsinventory` option to display all patch identifiers. To successfully rollback a patch, the patch identifier must be supplied.
invPtrLoc | Specifies the location of the `oraInst.loc` file. This command-line argument is needed when the `-invPtrLoc` command-line argument was used during installation. Oracle recommends the use of the default Central Inventory for a platform.
jdk | Specifies the location of a particular JDK (jar) to use instead of the default location under the Oracle home directory.
jre | Specifies the location of a particular JRE (Java) to use instead of the default location under the Oracle home directory.
local | Specifies that the OPatch utility rollback and update the local node and update the inventory of the local node. It does not propagate the patch or inventory update to other nodes. This command-line argument can be used on Oracle Real Application Clusters environments and non-clustered environments. If an entire cluster is shutdown before patching, then this argument can be used for non-rolling patches.
local_node | Specifies to the OPatch utility that this is the local node for the cluster. This command-line argument can be used on Oracle Real Application Clusters environments.
no_sysmod | Specifies that the OPatch utility need not update the files in the system. It will only update the inventory.
no_relink | This option does not perform any make operation in the patch. It can be used during multiple patch removal and to perform the compilation step only once.
oh | Specifies the Oracle home directory to use instead of the default directory.
opatch_post_end | Marks the end of the `post` option. This command-line argument is used with the `post` command-line argument. If this argument is not used, then everything after `post` is passed into `post`.
opatch_pre_end | Marks the end of the `pre` options. This command-line argument is used with the `pre` command-line argument. If this argument is not used, then everything after `pre` is passed into `pre`.
ph | Specifies the valid patch directory area. The utility will use the command types found in the patch directory to identify which commands are used for the current operating system.
post | Specifies the parameters to be passed inside the `post` script besides the standard parameters.
pre | Specifies the parameters to be passed inside the `pre` script besides the standard parameters.
remote_nodes | Specifies to the OPatch utility the list of remote nodes. This command-line argument can be used on Oracle Real Application Clusters environments.
report | Prints the action to the screen without executing it.
retry | Specifies how many times the OPatch utility should try in case of an inventory lock failure.
silent | Suppresses user interaction, and defaults any answers to "yes". Real Application Clusters setup does not support this option.
verbose | Prints output to the screen as well as to the log file.

### version Option

The `version` option shows the current version number of the OPatch utility. The following syntax is used for this option:

    <Path_to_OPatch>/opatch version

## Real Application Clusters Patching

A Real Application Clusters environment is the one in which active instances can concurrently execute transactions on a shared database. Patching in a Real Application Clusters environment is slightly different compared to patching a single node.

Interim Patching using OPatch follows a similar approach as that performed by Oracle Universal Installer to detect Oracle home and nodes of a cluster. OPatch interacts with the Oracle Universal Installer inventory through the Oracle Universal Installer Java SDK. If OPatch detects a cluster, it queries the inventory through Oracle Universal Installer to find the local node name and node list. If your node list is not updated, you can update it by using the `-updateNodeList` flag of Oracle Universal Installer. You can bypass remote actions using the `-local` flag.

    $ORACLE_HOME/oui/bin/<runInstaller or setup.exe> -updateNodeList ORACLE_ HOME=<oracle home
     location> CLUSTER_NODES=<commaseparated nodelist for example:node1,node2,node3>
    -noClusterEnabled

If you want to specify to OPatch the local node or remote nodes of a Real Application Clusters setup, you can use the `LOCAL_NODE` or `REMOTE_NODES` session variable and specify the node name(s).

    $ORACLE_HOME/oui/bin/<runInstaller or setup.exe> ORACLE_HOME=<oracle home location>
    REMOTE_NODES <commaseparated nodelist for example:node1,node2,node3> LOCAL_NODE=<nodelist
    for example:node1>

If OPatch does not automatically detect a Real Application Clusters or its nodes, you need to investigate into the contents of the inventory and ensure that they are complete.

### Types of Real Application Clusters Patching

Real Application Clusters can be patched in three different ways:

- All Node Patching

- Rolling Patching

- Minimum Downtime Patching

#### All Node Patching

In All Node Patching, all the nodes in the Real Application Clusters are initially brought down and the patch is applied on all the nodes. Then all the nodes are brought back up. This mode is normally used for very critical patches and it leads to maximum downtime. OPatch uses this mode as default for patch application, unless specified otherwise. The All Node Patching is explained with the help of an example below:

Figure 8-2 All Node Patching

Consider a Real Application Clusters setup as shown in Figure 8-2, "All Node Patching". Systems A,B, and C are nodes in this Real Application Clusters. When you perform an All Node Patching in this cluster, systems A,B, and C are brought down and patches are applied to all these nodes. Then systems A,B, and C are brought back up again.

#### Rolling Patching

In Rolling Patching, each node is shutdown, the patch applied and brought back up again. This is done node by node separately until all nodes in the Real Application Clusters are patched. This is the most efficient mode of applying an Interim patch to a Real Application Clusters setup because this results in zero downtime. Only some patches can be applied in this mode. The type is generally specified in the patch metadata. The Rolling Patching is explained with the help of an example below:

Figure 8-3 Rolling Patching

Consider a Real Application Clusters setup as shown in Figure 8-3, "Rolling Patching". Systems A,B, and C are nodes in this Real Application Clusters. When you perform a Rolling Patching in this cluster, system A is initially brought down and patch applied to it. Then it is brought up back. Then system B is brought down and patch applied and brought up. This is repeated for system C also. The patch is applied in a rolling fashion. The main advantage of this type of patching is that there is absolutely no downtime during the application of patches as only one system is brought down at any given time.

#### Minimum Downtime Patching

In Minimum Downtime Patching, the nodes are divided into sets. Initially, the first set is shut down and the patch is applied to it. After this, the second set is shut down. The first set is brought up and patch is applied to the second set. The second set is also brought up now. All the nodes in the Real Application Clusters are now patched. This mode leads to less downtime for the Real Application Clusters when both the sets are brought down. This mode is executed by using `-minimize_downtime` command line option. You can also activate this option from the response file. The Minimum Downtime Patching is explained with the help of an example below:

Figure 8-4 Minimum Downtime Patching

Consider a Real Application Clusters setup as shown in Figure 8-4, "Minimum Downtime Patching". Systems A,B, and C are nodes in this Real Application Clusters. They are divided into two sets: set 1 containing systems A and B and set 2 containing system C. When you perform a Minimum Downtime Patching in this cluster, set 1 is shutdown and patch is applied to it. Now, set 2 is shutdown. Set 1 is brought up and patch is now applied to set 2. After application of patch, set 2 is brought back up again. Now, both sets 1 and 2 are patched.

## About Patch Conflicts

All patches may not be compatible with one another. For example, if a patch has been applied, all the bugs fixed by that patch could re-appear once another patch is applied. This is called a conflict situation. OPatch detects such situations and raises an error when a conflict is detected.

### Types of Conflicts

OPatch can detect the following types of conflicts:

- Bug Superset: If all the bugs fixed by a patch in the system are also fixed by the patch to be applied, then this patch (the patch to be applied) is considered to be a superset of the patch already applied. If a bug superset condition is detected, it is not considered an error situation. All the subset patches are removed from the system and the new patch is applied.

  For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown in Figure 8-5. If you apply a patch E that fixes bugs 5,6,7,8,9, and 10 then patch E will be the superset of patch C and D.

  Figure 8-5 Bug Superset

  If you want OPatch to error out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory, you can use the `-no_bug_superset` flag.

  $ OPatch/opatch apply -no_bug_superset <Path_To_Patch>

  The following is a sample output of the message you would get when you use the `-no_bug_superset` flag:

      Oracle interim Patch Installer version 10.2.0.0.0
      Copyright (c) 2005, Oracle Corporation.  All rights reserved..
      Oracle Home       : /home/oracle_TEST/product/10.2.0/db_1
      Central Inventory : /home/OUIHome_Opatch
      from           : /home/oracle_TEST/product/10.2.0/db_1/oraInst.loc
      OPatch version    : 10.2.0.0.0
      OUI version       : 10.2.0.0.0
      OUI location      : /home/oracle_TEST/product/10.2.0/db_1/oui
      Log file location : /home/oracle_TEST/product/10.2.0/db
      _1/cfgtoollogs/opatch/opatch-2005_May_25_14-03-33-IST_Wed.log
      ApplySession applying interim patch '102000' to OH '/home/oracle
      _TEST/product/10.2.0/db_1'
      Apply Session failed: ApplySession failed to prepare the system. Interim patch
       102000 is a superset of the patch(es) [ 102000 ] in OH /home/oracle
      _TEST/product/10.2.0/db_1
      System intact, OPatch will not attempt to restore the system
      OPatch failed with error code 73

- Bug Conflict: If a set of bugs to be fixed by the current interim patch includes some but not all bugs already fixed by one or more previously installed interim patches it is called a bug conflict. You must remove the bug conflict before you proceed with the patching by using the `apply` command with `-force` flag, that rolls back the conflicting patches before applying the new one.

  For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown in Figure 8-6. If you apply a patch E that fixes bugs 1,3,5,7,9, and 10, you will find that this patch has fixed bugs 1,3,5,7,9, and 10, but has opened bugs 2,4,6, and, 8. This is a conflict situation.

  Figure 8-6 Bug Conflict

- File Conflict: If a set of files to be patched by the current interim patch include files already patched by one or more previously installed interim patches and it is not a bug superset, it is called a file conflict. You must remove the file conflict before you proceed with the patching by using the `apply` command with `-force` flag, that rolls back the conflicting patches before applying the new one.

- Combination Conflict: If a set of patches has a combination of bug superset, and bug or file conflict, it is called a Combination Conflict. It is an error situation. In this case, OPatch removes all conflicting patches as well as the subset patches and then re-applies the new patch.

  For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown in Figure 8-7. Patch C is the subset of patch D. Patch A and patch B are conflicting patches of patch D. If you apply this patch D that fixes bugs 1,3,5,6,7, and 8 with `-force` flag, you will find that OPatch would have rolled back patches A, B, and C and would have applied patch D.

  Figure 8-7 Combination Conflict

You can use the `-silent, -force`, and `-no_bug_superset` options in Real Application Clusters and their expected behavior is listed in the form of the following truth table:

Table 8-6 Truth Table

-silent | -force | -no_bug_superset | Expected behavior
--- | --- | --- | ---
N | N | N | OPatch prompts questions and applies the patches depending upon your responses.
N | N | Y | OPatch errors out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory. If it is a conflict case, OPatch prompts questions and applies the patches depending upon your responses.
N | Y | N | OPatch prompts questions and applies the patches depending upon your responses.
N | Y | Y | OPatch errors out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory. If it is a conflict case, OPatch prompts questions and applies the patches depending upon your responses.
Y | N | N | OPatch rolls back and applies the patch if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory. If it is a conflict case, OPatch errors out.
Y | N | Y | OPatch errors out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory. If it is a conflict case also, OPatch errors out.
Y | Y | N | OPatch rolls back all the patches and reapplies them without warning, even if the current patch bugs-to-fix is a superset, or a conflict, or the same as an installed patch bugs-fixed in the Oracle home directory.
Y | Y | Y | OPatch errors out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory. If it is a conflict case, OPatch rolls back and applies the new patch.

### Patch Conflict Detection and Resolution

OPatch detects and reports any conflicts encountered when applying an Interim patch with a previously applied patch. The patch application fails in case of conflicts. The `-force` option of OPatch can be used to override this failure. If `-force` is specified, the installer will first rollback any conflicting patches and then proceed with the installation of the desired interim patch.

You may come across a bug conflict and might want to remove the conflicting patch. This process is known as patch rollback. During patch installation, OPatch saves copies of all the files that were replaced by the new patch before the new versions of these files are loaded and stores it in $ORACLE_HOME/.patch_storage. These saved files are called rollback files and are the key to making patch rollback possible. When you rollback a patch, these rollback files are restored to the system. You should only override the default behavior by using the `-force` flag, if you have gained a complete understanding of the patch rollback process. To rollback a patch, execute the following command:

$ OPatch/opatch rollback -id <Patch_ID>

### Restoring Oracle Homes

Every time you apply a patch, you make changes to your inventory. Sometimes that change may corrupt the inventory. You can use the restore.sh or restore.bat script that comes with OPatch to remove any changes that were made to the inventory after the application of the patch.When you apply a patch, OPatch creates a snapshot of your inventory and stores it in $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory. For more information refer to "Interim Patch Versions".

When your inventory is corrupted, you need to perform the following steps to bring the application back to it's last known good state:

1. Ensure that the environment variable ORACLE_HOME is set properly.

2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

       For UNIX:
       $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

       For Windows:
       $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

       /bin/sh make.txt

## Logging and Tracing

Logging and Tracing is a common aid for debugging. OPatch maintains logs for all apply, rollback, and lsInventory operations. The log files are located in <ORACLE_HOME>/cfgtoollogs/opatch directory. Each log file will be tagged with the timestamp of the operation. Log files are named as opatch_<date mm-dd-yyyy>_<time hh-mm-ss>.log. Each time OPatch is executed a new log file is created.

For example, if a log file is created on May 17th, 2005 at 11.55 PM, then it will be named as follows:

    opatch_05-17-2005_23-55-00.log

Note: You can set OPatch to debug mode by setting the environment variable OPATCH_DEBUG to TRUE.

OPatch also maintains an index of the commands executed with OPatch and the log files associated with it in the `history.txt` file located in <ORACLE_HOME>/cfgtoollogs/opatch directory. A sample of the `history.txt` file is as follows:

    Date & Time : Tue Apr 26 23:00:55 PDT 2005
    Oracle Home : /private/oracle/product/10.1.0/db_1/
    OPatch Ver. : 10.2.0.0.0
    Current Dir : /scratch/oui/OPatch
    Command     : lsinventory
    Log File    :
    /private/oracle/product/10.1.0/db_1/cfgtoollogs/opatch/opatch-2005_Apr_26_23-00-55-PDT_Tue.log

### Levels of Logging

OPatch follows the Oracle Diagnostic Logging (ODL) Guidelines. You can set the log level by using the -logLevel <level> option available. This option is used to control the amount of logging OPatch performs, according to the ODL guidelines.

OPatch supports the following log levels:

- SEVERE

- WARNING

- INFO

- CONFIG

- FINE

- FINER

- FINEST

## Recovering from a Failed Patching Session Using OPatch

During patching, updates can occur in two phases:

1. System Update: In this phase, the files get replaced in the Oracle home.

2. Inventory Update: In this phase, the details of the patch applied is recorded in the inventory.

The following lists the various scenarios and how you recover from a failed patching session:

When you apply or rollback a patch, you get an interim inventory update exception.
  Cause: This occurs when the files on the system are patched, but the inventory update has failed. This may be due to a corrupted inventory.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command.

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

When you apply or rollback a patch, you get an OiiOneoffException exception.
  Cause: This occurs when the files on the system are patched, but the inventory update has failed. This may be because the base component of the interim patch may not be present in the inventory.
  Action: OPatch tries to restore the Oracle home automatically and gives a message for the same. If OPatch does not give a message stating that it has restored the Oracle home, perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command.

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

When you apply a patch and execute opatch lsinventory, it returns nothing.
  Cause: This may be due to losing all the patches applied before the application of the current patch or the patches might not have been updated in the inventory.
  Action: Perform the following steps:

  1. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  2. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

  3. If the files are properly patched, but the information is not updated in the inventory, then execute the following command:

         $ORACLE_HOME/OPatch/opatch apply -no_sysmod <Path_To_Patch>

     Ensure that the patch has been applied and have been recorded properly in the inventory by executing the following command:

         $ORACLE_HOME/OPatch/opatch lsinventory -detail

  4. If the files are still not patched properly, but you are able to see the patch in the lsinventory flag, you need to re-apply the patch using the no_inventory flag:

         $ORACLE_HOME/OPatch/opatch apply -no_inventory <Path_To_Patch>

When you apply a patch and execute opatch lsinventory, it does not return the details of the patch applied.
  Cause: This may be because OPatch may not have recorded the details of this patch in the inventory.
  Action: Perform the following steps:

  1. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  2. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

  3. If the files are properly patched, but the information is not updated in the inventory, then execute the following command:

         $ORACLE_HOME/OPatch/opatch apply -no_sysmod <Path_To_Patch>

     Ensure that the patch has been applied and have been recorded properly in the inventory by executing the following command:

         $ORACLE_HOME/OPatch/opatch lsinventory -detail

When you press Ctrl + C during the application or rollback of a patch and execute opatch lsinventory, it does not return the details of the patch applied or rolled back.
  Cause: This may be because OPatch might have stopped the application or rollback of the patch on pressing Ctrl + C.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command, if it is available.

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

When you apply a patch, you quit when OPatch failed to relink and prompted to continue.
  Cause: This may be due to relink failure.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command.

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. Resolve the re-link failure issue by ensuring that you are able to invoke make manually on a UNIX shell. After this, apply the patch again.

### Real Application Clusters Setup

When I apply a patch on a Real Application Clusters setup, and execute 'opatch lsinventory' on the local node, the patch is not listed.
  Cause: This may occur if OPatch failed to update inventory.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly in all the nodes of the cluster.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory of each node in the cluster and execute the restore command as follows:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) in each node of the cluster as follows:

         /bin/sh make.txt

  4. Apply the patch in each node in the cluster using the local flag:

         $ORACLE_HOME/OPatch/opatch apply -local <Path_To_Patch>

     Note: Ensure that all the nodes use the same OPatch version.

When I apply a patch on a Real Application Clusters setup, and execute 'opatch lsinventory' on the local node, it returns nothing.
  Cause: This may be because you might have lost all the patches applied earlier.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command in each node in the cluster.

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) in each node as follows:

         /bin/sh make.txt

  4. Apply the patch in each node using the local flag:

         $ORACLE_HOME/OPatch/opatch apply -local <Path_To_Patch>

     Note: Ensure that all the nodes use the same OPatch version.

When I rollback a patch on a Real Application Clusters setup, and execute 'opatch lsinventory' on the local node, it shows that the patch was not removed.
  Cause: This may occur if OPatch failed to update inventory.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory in each node in the cluster and execute the restore command as follows:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) in each node in the cluster as follows:

         /bin/sh make.txt

  4. Rollback the patch in all the nodes in the cluster using the local flag:

         $ORACLE_HOME/OPatch/opatch rollback -local -id <Patch_ID>

     Note: Ensure that all the nodes use the same OPatch version.

When I rollback a patch on a Real Application Clusters setup, and execute 'opatch lsinventory' on the local node, it returns nothing.
  Cause: This may be because you might have lost all the patches applied earlier.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command in each node in the cluster:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt

  4. Rollback the patch in the local node using the local flag:

         $ORACLE_HOME/OPatch/opatch rollback -local -id <Patch_ID>

  5. Rollback the patch on the other nodes also using local flag.

     Note: Ensure that all the nodes use the same OPatch version.

When I apply a patch on a Real Application Clusters setup, the patching in one node is fine (both the files and the inventory are fine), but I am not sure about the other nodes.
  Cause: This may be due to a failed system or inventory update.
  Action: Perform the following steps:

  1. Copy the Oracle home from the node that is fine to the other nodes.

  2. After copying the Oracle home, make sure that the `ORACLE_HOME/inventory/ContentsXML/comps.xml` file has the latest timestamp.

     Note: On Unix, use `touch` to change the timestamp.

  3. Update the nodes of the cluster. For more information on updating the nodes of the cluster refer to section "Updating the Nodes of a Cluster".

  4. Ensure that all the prerequisite checks listed in the section "Pre-requisite Checks in OPatch" pass.

When I apply a patch on a Real Application Clusters setup, the patching in one node is fine, but when I execute 'opatch lsinventory' on the other nodes, the patch is not listed.
  Cause: This may be due to a failed system or inventory update.
  Action: Perform the following steps:

  1. Copy the `ORACLE_HOME /inventory` directory from the node that is fine to the other nodes.

  2. After copying the `ORACLE_HOME /inventory` directory, make sure that the `ORACLE_HOME/inventory/ContentsXML/comps.xml` file has the latest timestamp.

     Note: On Unix, use `touch` to change the timestamp.

  3. Update the nodes of the cluster. For more information on updating the nodes of the cluster refer to section "Updating the Nodes of a Cluster".

  4. Ensure that all the prerequisite checks listed in the section "Pre-requisite Checks in OPatch" pass.

When I apply or rollback a patch on a Real Application Clusters setup, I am not able to apply or rollback the patch on all nodes.
  Cause: This may occur if the nodes are not properly updated.
  Action: Perform any one or more of the following:

  - Ensure that all the nodes in the cluster are up-to-date. If they are not, update the nodes of the cluster. For more information on updating the nodes of the cluster refer to section "Updating the Nodes of a Cluster".

  - Execute the appropriate command on all nodes of the cluster as follows:

        opatch apply -local [patch_location]

        opatch rollback -local [patch_location]

  - Execute the appropriate command on the local node of the cluster as follows:

        opatch apply [-local_node (node_name)] [-remote_nodes (comma separated node_names)]

        opatch rollback [-local_node (node_name)] [-remote_nodes (comma separated node_names)]

## Environment Variables Used by OPatch

OPatch uses some environment variables as follows:

ORACLE_HOME - This is the Oracle home location.

OPATCH_DEBUG - This is the log level that specifies the amount of logging OPatch should perform.

OPATCH_PLATFORM_ID - This is the unique platform ID.

PATH - This is the path information.

## Troubleshooting OPatch

This section provides solutions to errors that may occur during patch application.

- Not a valid patch area

- OPatch cannot find system commands like fuser, make

- Unable to remove a partially-installed interim patch

Not a valid patch area
  Cause: The directory that the OPatch utility is using to do the patch does not match the template for what it is checking. This can also occur when the utility is run from an invalid ship home directory.
  Action: When starting the OPatch utility, the directory needs the following:

  - A `/etc` directory that has the metadata files.

  - A `/files` directory that has the payload files.

  - The `/etc/config/inventory` file and the actions file under the same directory.

  If you did not start the OPatch utility from the `patch_id` directory, then you can use the following command:

      opatch apply /<Patch_Shiphome>

OPatch cannot find system commands like fuser, make
  Cause: The OPatch utility uses `fuser` on UNIX systems to check for active Oracle instances. On certain hp-ux systems, only a super-user can run `fuser`.
  Action: Do the following steps to resolve this problem:

  1. Set `/tmp` in your PATH.

     For more information refer to section "Check for System Commands".

  2. Create an empty file named `fuser`.

  3. Shut down the Oracle instances.

  4. Run the OPatch utility.

     Caution: Another way to resolve this problem is to give executable permission to other users for `fuser`. However, this exposes a potential security hole in the system, and is not recommended.

Unable to remove a partially-installed interim patch
  Cause: Interruption in the patching process is the potential cause for this problem. This may occur if you press CTRL+C during the patching process. If the error is the one that OPatch detects, it automatically takes care of it.
  Action: Perform the following steps:

  1. Ensure that the environment variable ORACLE_HOME is set properly.

  2. Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command as follows:

         For UNIX:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

         For Windows:
         $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

  3. On UNIX, source `$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt` file (if available) as follows:

         /bin/sh make.txt